Learning Compressible Models

Regularization is a principled way to control model complexity, prevent overfitting, and incorporate ancillary information into the learning process. As a convex relaxation of l0norm, l1-norm regularization is popular for learning in high-dimensional spaces, where a fundamental assumption is the sparsity of model parameters. However, model sparsity can be restrictive and not necessarily the most appropriate assumption in many problem domains. In this paper, we relax the sparsity assumption to compressibility and propose learning compressible models: a compression operation can be included into l1-regularization and thus model parameters are compressed before being penalized. We concentrate on the design of different model compression transforms, which can encode various assumptions on model parameters, e.g., local smoothness, frequency-domain energy compaction, and correlation. Use of a compression transform inside the l1 penalty term provides an opportunity to include information from domain knowledge, coding theories, unlabeled data, etc. We conduct extensive experiments on brain-computer interface, handwritten character recognition, and text classification. Empirical results show significant improvements in prediction performance by learning compressible models instead of sparse models. We also analyze the model fitting and learned model coefficients under different compressibility assumptions, which demonstrate the advantages of learning compressible models instead of sparse models. 1. Learning Compressible Models Since the introduction of lasso (Tibshirani, 1996), l1-regularization has become very popular for learning in high-dimensional spaces. A fundamental assumption of l1-regularization is the sparsity of model parameters, i.e., a large fraction of coefficients are zeros. This assumption might be too restrictive and not necessarily appropriate in some application domains. However, many signals in the real world (e.g., images, audio, videos) are found to